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Abstract 

This paper considers the performance of (j, fc)-regular low-density parity-check (LDPC) codes with message- 
passing (MP) decoding algorithms in the high-rate regime. In particular, we derive the high-rate scaling law for MP 
decoding of LDPC codes on the binary erasure channel (BEC) and the g-ary symmetric channel (g-SC). For the BEC, 
the density evolution (DE) threshold of iterative decoding scales like 0(fc~^) and the critical stopping ratio scales 
like Q{k~-'^^-'~^^). For the g-SC, the DE threshold of verification decoding depends on the details of the decoder 
and scales like Q{k~^) for one decoder 

Using the fact that coding over large finite alphabets is very similar to coding over the real numbers, the analysis 
of verification decoding is also extended to the the compressed sensing (CS) of strictly-sparse signals. A DE based 
approach is used to analyze the CS systems with randomized-reconstruction guarantees. This leads to the result that 
strictly-sparse signals can be reconstructed efficiently with high-probability using a constant oversampling ratio (i.e., 
when the number of measurements scales linearly with the sparsity of the signal). A stopping-set based approach is 
also used to get stronger (e.g., uniform-in-probability) reconstruction guarantees. 

Index Terms 

LDPC codes, verification decoding, compressed sensing, stopping sets, q-ary symmetric channel 

I. Introduction 

Compressed sensing (CS) is a relatively new area of signal processing that has recently received a large amount 
of attention. The main idea is that many real-world signals (e.g., those sparse in some transform domain) can 
be reconstructed from a relatively small number of linear dot-product measurements. Its roots lie in the areas of 
statistics and signal processing [1], [2], [3], but it is also very much related to previous work in computer science 
[4] and applied mathematics [5], [6], [7]. CS is also very closely related to error correcting codes, and can be seen 
as source coding using linear codes over real numbers [8], [9], [10], [11], [12]. 

In this paper, we analyze the performance of low-density parity-check (LDPC) codes with verification decoding 
[13] as applied to CS. The resulting decoding algorithm is almost identical to that of Sudocodes [8], but a more 
suitable code ensemble is chosen and a more precise analysis is presented. Since most of the interesting applications 
of CS require very sparse (or compressible) signals, the natural mapping to coding implies high-rate codes. One 
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new characteristic of this analysis, which is also interesting from a coding perspective, is that the performance 
estimates hold uniformly as the code rate approaches one. This allows us to explore the sparsity (or rate) regime 
that makes sense for compressed sensing. 

An important implication of this work is that our randomized reconstruction system allows linear-time reconstruc- 
tion of strictly-sparse signals with a constant oversampling ratio. In contrast, all previous reconstruction methods 
with moderate reconstruction complexity have an oversampling ratio which grows logarithmically with the signal 
dimension. 

A. Background on LDPC Codes 

LDPC codes are linear codes introduced by Gallager in 1962 [14] and re-discovered by MacKay in 1995 [15]. 
Binary LDPC codes are now known to be capacity approaching on various channels when the block length tends 
to infinity. They can be represented by a Tanner graph, where the i-th variable node is connected to the j-th check 
node if the entry on the i-th column and j-th row of its parity-check matrix is non-zero. 

LDPC codes can be decoded by an iterative message-passing (MP) algorithm which passes messages between 
the variable nodes and check nodes iteratively. If the messages passed along the edges are probabilities, then the 
algorithm is also called belief propagation (BP) decoding. The performance of the MP algorithm can be evaluated 
using density evolution (DE) [16] and stopping set analysis [17] [18]. Each method provides a decoding threshold 
for the code ensemble. 

B. Structure of the Paper 

SectionlUprovides background information on coding and CS. Sectionlllll summarizes the main results. In Section 
II VI proofs and details are given for the main results based on DE. While in Section |V] proofs and details are provided 
for the main results based on stopping-set analysis. Section [Vll discusses a simple information-theoretic bound on 
the number of measurements required for reconstruction. Section IVIII presents simulation results comparing the 
algorithms discussed in this paper with a range of other algorithms. Finally, some conclusions are discussed in 
Section IVnl] 

[Author's Note: The equations in this paper were originally typeset for two-column presentation, but we have 
submitted it in one-column format for easier reading. Please accept our apologies for some of the rough looking 
equations.] 

II. Background on Coding and CS 

A. Encoding and Decoding 

The sparse graph representation of LDPC codes allows encoding and decoding algorithms to be implemented 
with linear complexity in the code length n. Since LDPC codes are usually defined over the finite field GF{q) 
instead of the real numbers, we need to modify the encoding/decoding algorithm to deal with signals over real 
numbers. Each entry in the parity-check matrix is either or a real number drawn from a continuous distribution. 
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The parity-check matrix $ e T^mxn full-rank with high probability (w.h.p.) and is used as the measurement 

matrix in the CS system (e.g., the signal vector x E M" is observed as y — ^x). 

The process of generating the observation symbols can also be seen in a bipartite Tanner graph representation. 
Figure. 1 shows the encoder structure. Each non-zero entry in <i> is the edge-weight of its corresponding edge in 
this graph. Therefore, the observation process associated with a degree d check node is as follows: 

1) Encoding: The observation symbol is the weighted sum (using the edge weights) of the d neighboring signal 
components. 

In this work, we only consider strictly sparse signals and we use two decoders based on verification which were 
proposed and analyzed in [13]. The second algorithm was also proposed independently for CS in [8]. The decoding 
process uses the following rules: 

1) If a measurement is zero, then all the neighboring variable nodes are verified as zero. 

2) If a check node is of degree one, verify the variable node with the value of the measurement. 

3) [Enhanced verification] If two check nodes overlap in a single variable node and have the same measurement 
value, then verify that variable node to the value of the measurement. 

4) Remove all verified variable nodes and the edges attached to them by subtracting out the verified values from 
the measurements. 

5) Repeat steps 1-4 until decoding succeeds or makes no further progress. 

Note the first algorithm follows steps 1, 2, 4 and 5. The second algorithm follows steps from 1 to 5. These two 
algorithms correspond to the first and second algorithms in [13] and are referred to as LMl and node-based LM2 
(LM2-NB) in this papejjl Note that LDPC codes with regular check degree and Poisson symbol degree with LM2- 
NB decoding is identical to the Sudocodes introduced in [8]. In [20], the LM2-NB algorithm which is an enhanced 
version of message-based LM2 (LM2-MB) is analyzed precisely. 

'in [13], the second algorithm (which we refer to as LM2) was described in a node-based (NB) fashion (as above), but analyzed using 
a message-based (MB) density-evolution. There is an implicit assumption that the two algorithms perform the same. In fact, they perform 
differently and the LM2-NB algorithm is superior as observed in [19] [20]. 
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In general, the scheme described above does not guarantee that all verified symbols are actually correct. The 
event that a symbol is verified but incorrect is called false verification (FV). In order to guarantee there are no 
FVs, one can add a constraint on the signal such that the weighted sum, of any subset of a check node's non-zero 
neighbors, does not equal to zero [9] [8]. Another approach (e.g., taken in this paper) is to consider random signals 
with continuous distributions so that FV occurs with probability zero. Finally, if the measured signal is assumed to 
be non-negative, then FV is impossible for the LMl and LM2-NB decoding algorithms. 

Verification decoding was originally introduced and analyzed for the q-SC. It is based on the observation that, 
over large alphabets, the probability that "two independent random numbers are equal" is quite small. This leads 
to the verification assumption that any two matching values (during decoding) are generated by the same set of 
non-zero coefficients. The primary cormection between CS, codes over real numbers, and verification decoding lies 
in the fact that: 

The verification assumption applies equally well to both large discrete alphabets and the real numbers. 
B. Analysis Tools 

Based on the bipartite graph structure, LDPC codes can be decoded efficiently using iterative MP algorithms. 
The average performance of MP decoding algorithms can be analyzed with density evolution (DE) [16] or extrinsic 
information transfer (EXIT) charts [21]. The concentration theorem [16] shows that random realizations of decoding 
are close to the average behavior w.h.p. as the block length goes to infinity. DE analysis provides a threshold below 
which decoding (or reconstruction) succeeds w.h.p.. The decoding threshold can be improved by optimizing the 
edge degree distribution (d.d.) pair \{x) and p{x). 

Decoding can also be analyzed using combinatorial methods such as stopping set analysis [17] and [18]. Stopping 
set analysis gives a threshold below which all error patterns can be recovered with certainty under the assumption 
of no FV. Note that DE and stopping set analysis lead to different thresholds in general. Since stopping set analysis 
implies uniform recovery of all the error patterns, instead of just most of them, the threshold given by stopping set 
analysis is always lower than the one given by DE. For example, DE analysis of (3, 6) regular codes on the BEC 
shows that erasure patterns with size less than 43% of the code length can be corrected w.h.p. [22]. On the other 
hand, the result from stopping set analysis guarantees that most codes correct all erasure patterns with size less 
than 1.8% of the code length when n — > oo. 

Likewise, in CS systems, there are two standard measures of reconstruction: uniform reconstruction and ran- 
domized (or non-uniform) reconstruction. A CS system achieves randomized reconstruction if most randomly 
chosen measurement matrices recover most of the signals in the signal set. While a CS system achieves uniform 
reconstruction if a measurement matrix and the decoder recover all the signals in the signal set with certainty. 
Another criterion, which is between uniform reconstruction and randomized reconstruction, is what we call uniform- 
in-probability reconstruction. A CS system achieves uniform-in-probability reconstruction if, for any signal in the 
signal set, most randomly chosen measurement matrices achieve successful decoding. 
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Since DE and the concentration theorem lead to w.h.p. statements for MP decoding over all signals and graphs, 
it is natural to adopt a DE analysis to evaluate the performance of randomized reconstruction CS systems based on 
LDPC codes. For uniform reconstruction, a stopping set analysis of the MP decoder is the natural choice. While 
this works for the BEC, the possibility of FV prevents this type of strong statement for verification decoding. If the 
non-zero entries of $ are chosen randomly from a continuous distribution, however, then the probability of FV is 
zero for all signals. Therefore, one can use stopping set analysis to analyze MP decoding of LDPC code ensembles 
and show that the LDPC codes with MP decoding achieves uniform-in-probability reconstruction in the CS system. 
The reader is cautioned that these results are somewhat brittle, however, because they rely on exact calculation and 
measurement of real numbers. 

Understanding CS systems requires one to consider how the system parameters (e.g., the number of measurements 
and the sparsity of the signal) scale in the regime where the signal is both high-dimensional and extremely sparse. 
To compare results, we focus on the oversampling ratio (i.e., the number of measurements divided by the number 
of non-zero elements in the signal) required for reconstruction. This leads us to a scaUng law approach to the 
standard DE and stopping set analysis. 

C. Decoding Algorithms 

In CS, optimal decoding (in terms of oversampling ratio) requires a combinatorial search that is known to be 
NP-Hard [23J. Practical reconstruction algorithms tend to either be based on Unear programming (e.g., basis pursuit 
(BP) [IJ) or low-complexity iterative algorithms (e.g., Orthogonal Matching Pursuit (OMP) |24|). A wide range 
of algorithms allow one to trade-off the oversampling ratio for reconstruction complexity. In [8J, LDPC codes are 
used in the CS system and the algorithm is essentially identical to the verification based decoding proposed in 
[13]. The scaling law analysis shows the oversampling ratio for LDPC codes based CS system can be quite good. 
Encoding/decoding complexity is also a consideration. LDPC codes have sparse bipartite graph representation so 
that encoding and decoding algorithms with complexity linearly with the code length can be developed. 

There are several existing MP decoding algorithms for LDPC codes over non-binary fields. In [22] and [25], an 
analysis is introduced to find provably capacity-achieving codes for erasure charmels under MP decoding. Metzner 
presents a modified majority-logic decoder in [26] that is similar to verification decoding. Davey and MacKay 
develop and analyze a symbol-level MP decoder over small finite fields [27]. Two verification decoding algorithms 
for large discrete alphabets are proposed by Luby and Mitzenmacher in [13] and called as LMl and LM2 in this 
paper. (Two capacity-achieving algorithms are presented by ShokroUahi and Wang in [28] and are denoted as SWl 
and SW2 in this paper.) The list-message-passing (LMP) algorithm [20] provides a smooth trade-off between the 
performance and complexity of SWl and SW2. These algorithms are summarized in [20]. 

One can get a rough idea of the performance of these algorithms by comparing their performance for the standard 
(3, 6)-regular LDPC code. A standard performance measure is the noise threshold (or sparsity threshold for CS) 
below which decoding succeeds with high probability. The threshold of the LMl algorithm in this case is 0.169 . 
This means that a long random (3, 6)-regular LDPC code will correct a q-SC error pattern with high probabiUty 
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Fig. 2. Thresholds vs l-R, where _R is the code rate, for LMl stopping set/DE analysis and the BEC stopping set analysis. 

as long as the error rate is less than 0.169. Likewise, it means that using the same code for LMl reconstruction 
of a strictly-sparse signal with succeed w.h.p. as long as the sparsity rate (i.e., fraction of non-zero elements) of 
the signal vector is less than 0.169. The LM2-MB algorithm improves this threshold to 0.210 and the LM2-NB 
algorithm further improves this threshold to 0.259 [20]. 

Likewise, the stopping-set analysis of the LMl algorithm in Section V shows that a (3, 6) -regular code exists 
where LMl succeeds (ignoring FV) for all error (or sparsity) patterns whose fraction of non-zero entries is less than 
0.0055. In comparison, the BEC stopping-set threshold of the (3, 6) code is 0.018 for erasure patterns. However, 
both of these thresholds can be increased significantly (for the same code rate) by increasing the variable node 
degree. In fact, the (7, 14)-regular LDPC code gives the best (both LMl and BEC) stopping-set thresholds and they 
are (respectively) 0.0364 and 0.0645. Finally, if the signal is non-negative, then FV is not possible during LMl 
decoding and therefore 0.0364 is a lower bound on the true LMl rate-i threshold for uniform reconstruction. Fig. 
|2] shows the decoding/recovery thresholds for BEC stopping set analysis, LMl stopping set analysis and LMl DE 
analysis. Note that if the signal coefficients are non-negative, the threshold of LMl given by stopping set analysis 
is comparable to the strong bound given in [29, Fig. 1(a)], and the threshold of LMl given by DE analysis is 
comparable to the weak bound given in [29, Fig. 1(b)]. 
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Since the scaling law analysis becomes somewhat tedious when complicated algorithms are applied, we consider 
only the (j, /c)-regular code ensemble and the relatively simple algorithms LMl and LM2-MB. The rather surprising 
result is that even with regular codes and simple decoding algorithms, the scaling law implies that LDPC codes 
with verification decoding perform very well in CS systems with strictly-sparse signals. 

D. Signal Model 

There are some significant differences between coding theory and CS. One of them is the signal model. The 
first difference is that coding theory typically uses discrete alphabets (see [30] for one exception to this) while 
CS deals with signals over the real numbers. Fortunately, some codes designed for large discrete alphabets (e.g., 
the g-ary symmetric channel) can be adapted to the real numbers. By exploring the connection and the analogy 
between real field and finite field with large g, the CS system can be seen as an essentially a syndrome-based 
source coding system [10]. Using the parity-check matrix of a non-binary LDPC code as the measurement matrix, 
the MP decoding algorithm can be used as the reconstruction algorithm. 

The second difference in the signal model is that CS usually models the sparse signal x e as a random 
vector in an iV-dimensional unit weak Ir ball which is denoted as U{£^j.). Note that the weak £r ball of radius 
M consists of vectors x whose decreasing rearrangement, denoted as x*, satisfies \xl\ > jxjl > • • • > with 
|a;*| < Mi^^/^. The constraint x G U{£^ ^) defines the approximate sparsity property of the signal. As r approaches 
zero, the decreasing rearrangement of the x coefficients must decay faster For decoding, the constants r and M 
provide a partial ordering of the set {x\x G U{£^q)} and allow one to regularize the ill-conditioned decoding 
problem. 

In information theory and coding theory, the most commonly used signal model is a probabilistic model; the 
signal is treated as a random variable and the pdf is used to describe the signal. The sparsity of the signal can also 
be captured in the probabilistic model. For example, we can use a weighted sum of a Dirac delta function at zero 
and a very wide uniform distribution to model a strictly sparse signal. 

E. Interesting Rate Regime 

In coding theory, the code rate depends on the application and the interesting rate regime varies from close to 
zero to almost one. In CS systems, the signal is sparse in some domain and becomes increasingly sparse as the 
dimension increases. Intuitively, this means we can use codes with very little redundancy or very high code rate 
to represent the signal. So the interesting rate regime for CS systems is the high-rate regime. We consider the 
relationship between the system parameters and how they scale as the rate goes to one. The answer lies in the 
scaUng law analysis developed in this paper. 

III. Main Results 

The main results of this paper are listed as follows. The details follow in section |IV] and section [V] Note that 
all results hold for randomly-chosen regular LDPC codes with variable-degree j and check-degree k. For a given 
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j, we can increase k and observe how the decoding threshold scales. This provides a scaling law for the threshold 
and leads to conditions for successful reconstruction and the converse. One consequence of this is that randomized 
reconstruction can be achieved, for CS of strictly-sparse signals, when the number of measurements scales linearly 
with the sparsity of the signal. 

(i) [DE-BEC] For the BEC, there is a iiT < cx3 such that: a check-regular LDPC codes with average variable 
node degree j > 2 and check-degree k can recover a 6 < ajj /{k — 1) fraction of erasures (w.h.p. as n ^ oo) 
when k > K. The constant aj (independent of fc) is essentially the fraction of the optimal S* = j /k achieved as 
n ^ oo and the rate goes to one. Conversely, if the erasure probability 6 > ctjj /{k — 1), then decoding fails (w.h.p. 
as n — > oo) for all k. 

(ii) [SS-BEC] For any < 6* < 1, there is a iiT < oo such that: for all k> K, a. {j, A:)-regular LDPC code with 
j > 3 can recover all erasures (w.h.p. as n ^ oo) of size 6ne [k — 1) 

(iii) [DE-g-SC-LMl] For the g-SC, when one chooses a code randomly from the (j, fc) regular ensemble with 
j > 2 and uses LMl as decoding algorithm, then there is a Ki < oo such that one can recover almost all 
error patterns of size n6 for 6 < aj{k — (w.h.p. as n — + oo) for all k > Ki. Conversely, when 
S > aj{k — there is a K2 < 00 such that the decoder fails (w.h.p. as n — > 00) for all k > K2- 

(iv) [DE-q-SC-LM2-MB] For the q-SC, when one chooses a code randomly from the {j, k) regular ensemble 
with J > 3 and uses LM2-MB as decoding algorithm, then there is a Ki < 00 such that one can recover almost 
all error patterns of size n5 for 6 < ctjj /k (w.h.p. as n ^ 00). The constant cij (independent of k) is essentially 
the fraction of the optimal 5* — j /k achieved as the rate goes to one. Conversely, there is a K2 < 00 such that 
the decoder fails (w.h.p. as n ^ 00) when 6 > ajj/k for all k > A'2. 

(v) [SS-g-SC-LMl] For any < < 1, there is a /\ < 00 such that: For all k> K,a {j, fc)-regular LDPC code 
with j >2> using LMl decoding can recover (w.h.p as n ^ 00) all q-SC error patterns of size 9n(3j{k — 

if no false verifications occur. 

Note that the constants K, Ki, K2 and ctj in (i), (ii), (iii), (iv) and (v) are different, but for the simphcity of 
expression, we use the same notation. 

The rest of the paper is organized as follows. In Section |IV] we derive the high-rate scaling based on DE 
analysis. We first show the high-rate scaling analysis for the BEC under MP decoding. Then, the analysis of high- 
rate scaling for the q-SC with LMl and LM2-MB decoding algorithms is shown. In Section |V] we derive the 
stopping set analysis for the q-SC with LMl decoding algorithm and the high-rate scaling. The simulation results 
are shown in Section IVIII 

IV. High-Rate Scaling Via Density Evolution 

A. DE Scaling Law Analysis for the BEC 

DE analysis provides an explicit recursion which connects the distributions of messages passed from variable 
nodes to check nodes at two consecutive iterations of MP algorithms. In the case of BEC, this task has been 
accomplished in [31] and [22], It has been shown that the expected fraction of erasure messages which are passed 
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in the i-th iteration, called Xi, evolves as Xi — 5\{1 — p(l — Xi^i)) where 6 is the erasure probability of the 
channel. For general channels, the recursion may be much more complicated because one has to track the general 
distributions which cannot be represented by a single parameter [32]. 

To illustrate the scaling law, we start by analyzing the BEC case using DE. Although this is not applicable to 
CS, it motivates the scaling law analysis for the g-SC which is related to CS. 

The scaling law of LDPC codes of check-regular ensemble over the BEC is shown by the following theorem. 

Theorem 1: Consider a sequence of check-regular LDPC codes with fixed variable node degree distribution X{x) 
and increasing check degree k. Let j = 1/ X{x)dx be the average symbol degree and aj, which is called a- 
threshold, be the largest a such that A (l — e^"^^) < x for x £ (0, 1]. For the erasure probability S = aj/(k — 1), 
the iterative decoding of a randomly chosen length-n code from this ensemble fails (w.h.p as rt ^ oo) for all k if 
a >ctj. Conversely, if a < cej, then there exists a K < oo such that iterative decoding succeeds (w.h.p as n ^ oo) 
for all k>K. 

Lemma 1: For all s > and fc^^" > the sequence Ofe = (l — prr)'^ is strictly increasing in k and 

1-xk-' <ak<e-''''~\ (1) 
Proof: [Proof of Lemma [l] We restrict our attention to a: > because the proof is simplified in this case and 
the continuation does not require x < 0. We show that ak is strictly increasing with k by considering the power 
series expansion of Ina/j, which converges if fc^+'* > \x\. This gives 

oo ^ 

= fcln (l - ^) = -^k-^ - E -mTT^' 

1=2 

and keeping only the first term shows that \nak < —xk^'^. Since all the terms are negative and decreasing with k, 
we see that ak is strictly increasing with k. Since ak is convex in x for fc^+* > \x\, the lower bound > 1 — xfc^" 
follows from tangent lower bound at a; = 0. ■ 
Proof: [Proof of Theorem [T] Using the change of variable, Xi = ^^Vi^ the DE recursion can be scaled to 

By Lemma [T] (1 — ■jf^)''^^ increases monotonically (for x < k — 1) to e~^, and we see that fk{y) decreases 
monotonically to /,((/) = (l — e^"^^^). If a > aj, then (by the definition of aj) f*{y) > y for some 
y e (0, 1]. Since fk{y) > f*{y), the recursion jy^+i — fk {yi) will not converge to zero (from yo = 1) and iterative 
decoding will fail for all k w.h.p. as n — > oo. 

If a < ctj, then < y for y E (0, 1]. Since fk{y) \ f*{y), we can choose iiT < oo to be the first k such 

that fk{y) < y for y e (0, 1]. In this case, the recursion j/^+i = fk (yi) will converge to zero (from yo — 1) for all 
k > K and iterative decoding will succeed w.h.p. as n — > oo. ■ 
The following proposition determines a few a-thresholds explicitly. 

Proposition 1: For (j, k) regular LDPC codes, the a-threshold is given by oij with 7x2 = 0.5, Ss ~ 0.8184, and 
a4 ~ 0.7722. 
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Proof: See Appendix U ■ 
Remark 1: For example, \f j — 2> and a — 0.75 < 0.8184 = a^, then numerical results show that K = 9 suffices 
so that DE converges for all A: > 9 when 5 < 3(0.75)/(fc - 1). 

B. Concentration Theorem 

Note that in the proof of Theorem [T] we make use of the standard concentration theorem [32] for MP decoding. 
For example, the "w.h.p." statements rely on the concentration theorem. However, the concentration theorem for 
the BEC DE analysis proved in [32] holds only for fixed k (i.e., k is independent of n). In many CS applications, 
the desired number of measurements is sublinear in the block length (i.e., k ~ n'^ with oj < 1). In this section, we 
discuss how the proof of the concentration theorem can be modified to handle this case. 

The concentration theorem [32] shows the performance of a randomly chosen code concentrates around its 
expected value so that 



Tie ^ 

where Z^^'> is the r.v. which is equal to the number of variable-to-check erasure messages in the i-th iteration, 
p(^) — E [Z*^^)] /ue is the expected erasure rate, Ue is the number of edges in the graph, and /? is a constant which 
depends on the degree distribution. Let T^_, be the number of edges in the depth-2£ directed neighborhood of the 
i-th edge, then the constant (3 can be chosen to be a/ max^ T^.i, where a is a constant independent of n and t. For 
regular ensembles, the number T^ i is independent of i and this implies /3 = (l/(j^^fc^^)) . Concentration occurs 
as n CX3 for fixed j, k and 

In this section, we consider the case where the check-node degree increases with n. In this case, the concentration 
theorem is not informative if k grows faster than n^/'^'^^\ But, the following theorem shows that the concentration 
theorem can be modified to handle the scaling law. 

Theorem 2: For check-degree k — n'^ and erasure rate 5 — the performance (in terms of the fraction of 
erasure messages) of a code randomly chosen from the (j, fc)-regular ensemble over BEC concentrates around its 
expected value exponentially in ni+^f where I is the number of decoding iterations. Mathematically, there exist 
constants f3'{j,uj,£) and N(j,uj,£) such that 



Pr 



nj 



>, <2e-/3'"^^^ (4) 



for any e > and n > N. 

Proof: See the Appendix HH ■ 

C. DE Scaling Law Analysis for the q-SC 

1) DE Scaling Law Analysis for LMl: For the simplicity of our analysis, we only consider (j, fc)-regular code 
ensemble and the LMl decoding algorithm [13] for the g-SC with error probability 5. The DE recursion for LMl 
is (from [13]) 



Xi 



+i=5[l- l-{l-5){l-{l-x,f-^) 



i 1 nfe-l\^"l 



(5) 
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where Xi is the fraction of unverified messages in the i-th iteration. Our analysis of the scaling law relies on the 
following lemma. 

Lemma 2: Let the functions gt+iix) and g^+iix) be defined by 



gk+iix) = — 1 
ai \ 



1-1 



1-1 ^ 



— lA J -1 — 



k\ 3-1 



ajX 



and 



— J — 1 1 — 1 
1 



where aj > 1, a € (0, ckj], and j > 2. For x e (0,1] and k > a-j , these functions satisfy (i) gkix) < 
gk{x), (ii) 5fe(a;) is monotonically decreasing with A; for fc > a^"^, and (iii) limk-^oo gk{x) — limfe„,oo 5fe(a;) = 

-3-1 7-i\J~i 



^ 1 - e-"^ 

aj \ 

Proof: See the Appendix Hill ■ 
Theorem 3: Consider a sequence of (j, /c)-regular LDPC codes with fixed symbol degree j > 2 and increasing 
check degree fc. Let aj be the largest a such that (1 — e~"^ ^)"'^^ !i 2; for x e (0, 1]. If the sparsity of the 
signal is n6 for 5 = a{k — l)^-J/(-J^i) and a < aj, then there exists a Ki such that by randomly choosing a 
length-n code from the {j, fc) regular LDPC code ensemble, LMl reconstruction succeeds (w.h.p as n ^ 00) for 
all fc > Ki. Conversely, if a > aj then there exists a A'2 such that LMl reconstruction fails (w.h.p as n ^ 00) 
for all k>K2. 

Proof: Scaling (|5]l using the change of variables S ~ aik — l)^-J/(-J^i) and Xi = ajyi{k — l)~J/(J"i) gives 
Vi+i ~ 9k [Ui)- The function 5^.(0;) also allows us to define the upper bound Zi+i — gj. {zi) where Zi < yi implies 
Zi+i < Vi+i- 

f \ k 

Since (l — |) increases monotonically to e^^, we see that g^iy) decreases monotonically to g*{y). If a < oij, 
then 5*(y) < y for all y e (0, 1]. Since gk{y) < Vkiu) \ 9*{y)^ we can choose Ki < cx) to be the first k such 
that 'gkiy) < y foi" y £ (0: !]■ Ill this case, the recursion z/i+i = g/c (j/i) will converge to zero (from yo — 1) for 
all k > Ki and iterative decoding will succeed w.h.p. as n ^ 00. 

If a > cij, then (by the definition of oij) g^.{y) > y for some y G (0, 1]. Since hnik^oo gk{y) = 9*{y), there 
exists a K2 such that, for all fc > A'2, the recursion y^+i = gk (yi) will not converge to zero (from yo = 1) and 
iterative decoding will fail w.h.p. as n ^ 00. ■ 

Remark 2: If a randomly chosen code from the {j, fc) regular ensemble is applied to a CS system with LMl 
reconstruction, then randomized reconstruction succeeds (w.h.p as n 00) when the sparsity is n6 with S < 
aj{k—l)~^^'^^^^K This requires m — jnS measurements with an oversampling ratio of 7 > 70 = a 5~^/^ j. 
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The following lemma shows how to calculate the constants in front of the scaling laws. 
Corollary 1: 0L2 = 1, « 1.87321, « 1.66455 and « 1.52073. 

Proof: See Appendix IIVI ■ 
Corollary 2: For regular LDPC codes and LMl reconstruction, choosing j — [in |] gives a uniform lower bound 
on the oversampling ratio (as (5 ^ 0) of [in |] e. 

Proof: The minimum oversampling ratio 70 — ' jS^^^^ < j6^^^^ and we choose j — [in j-] , taking the 
logarithm of both sides shows that 



I1170 < In 



■4 



^ In i < In 



[inir--<5 



■4 



(6) 



2) Scaling Law Analysis Based on DE for LM2-MB: For the second algorithm in [13], the DE recursion for the 
fraction Xi of unverified messages in the i-th iteration is 



X,+i = S\^X{1- p{l- Xi)) + X {1- p{l-X^)) 

(^p (1 - X,) ~ p(l - (1 - (5) A (1 - p (1 - X,)) - X,) 



Like the analysis of LMl, we first introduce a lemma to bound the scaled DE equation. 
Lemma 3: The functions gk{x) and gk{x) are defined as 



(7) 



9k( 



k 



1-1- 



1 _ 

^ k 



J-1 



fc— 1 

where s{x) = 1 - (1 - ^) , i.e., 1 - p(l - y), and 



fffc(a;) = 



1-1 



ajx 



ajx 



k 
k\ 



k\ J-i 



For X G (0, 1] and k > a, these functions satisfy (i) gk{x) > gk{x), (ii) linifc^oo 9kix) ~ limfc^oo gki^) — 9*{x) 
where 

g*{x) ^ ^ (1 - e-"J"-)^-' (1 + (j - 2)e-"^"-) , (8) 
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and (iii) gk{x) is a monotonically decreasing function of k. 
Proof: See the Appendix [Vl 



Theorem 4: Consider a sequence of (j, /c)-regular LDPC codes with variable node degree j > 3. Let oij be the 
largest a such that (l — e^"^^y ^ (l + (j — 2)e^"-'^) < x for x e (0, 1]. If the sparsity of the signal is nS with 
S — aj/k and a <aj, then there exists a Ki such that LM2-MB reconstruction succeeds (w.h.p as n oo) for 
all k > Ki. Conversely, if a > aj then there exists a K2 such that LM2-MB decoding fails (w.h.p as n ^ 00) for 
all k>K2. 

Proof: The LM2-MB DE recursion is given by (|7]i. Using the change of variables Xi = ^^TJi and S ~ 
the scaled DE equation can be written as i/i+i = gkiVi)- Taking the limit as fc — > 00 gives j/i+i ~ g*{yi)- 

If a < ctj, then the definition of aj impUes that g*{y) < y for y £ (0, 1]. Since gk{y) < Vkiy) \ 9*{y) (by 
Lemma [3]), we can choose if 1 < 00 to be the first k such that gj,(y) < y for y e (0, 1]. In this case, the recursion 
2/j+i — 9k {yi) will converge to zero (from yo — 1) for all k > Ki and iterative decoding will succeed w.h.p. as 
n "> 00. 

If a > ttj, then (by the definition of Ixj) g^,{y) > y for some y G (0, 1]. In this case, there is a K2 and y such 
that gk{y) > y for all k > K2, and the recursion y^+i = gk {yi) does not converge to zero (from yo — 1) and 
iterative decoding will fail w.h.p. as n ^ 00. 

For j = 2, the quantity a2 is undefined because (l — e^^^^Y " (l + {j — 2)e^"-'^) = 1. This implies that 
(2, fc) regular LDPC codes do not obey this scaling law for LM2-MB decoding. ■ 

Remark 3: If a randomly chosen code from the {j, k) regular ensemble is applied to a CS system with LM2-MB 
reconstruction, then randomized reconstruction succeeds (w.h.p as n ^ 00) when the sparsity is n6 with d < ajj/k. 
This requires m > jnS measurements and an oversampling ratio of 7 > 70 = l/ofj. 

Remark 4: For (j, fc) regular LDPC codes, the a-threshold of LM2-MB is given by aj and can be calculated 
numerically to get Sa = i, 54 w 0.34 and w 0.37. 

The interesting part of this result is that the number of measurements needed for randomized reconstruction with 
LM2-MB scales linearly with the sparsity of the signal. All previous reconstruction methods with reasonable 
complexity require a super- linear number of measurements. 

D. Concentration Theorem 

Note that the proof of Theorem [3] also depends on the standard concentration theorem in [32]. Like the BEC 
case, the result becomes non-informative if fc grows faster than n^/*^^^). Therefore, we conjecture (i.e., give without 
proof) the following concentration result for LMl decoding for a randomly chosen code and q-SC error pattern. 
For k = n'^ and S — a{k — l)^^/(J^i\ the fraction of unverified messages after £ iterations concentrates around 
its expected value p^^K Mathematically, there exist constants f3'{j,uj, ,£) and N{j,uj,£) such that 




for any e > and n > N. 
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To show the concentration theorem, the idea is similar with the one used in the BEC case. We first reduce the 
graph to a smaller one by removing the verified edges in the first iteration, then prove the concentration for the 
residual graph. The probability that an edge is not removed in the first iteration is S + {I — S) ((1 — (1 — (5)*^^^)"' , 
so the average number of edges which are not removed in the first iteration is ^(5 + (1 — (5) ((1 — (1 — 5)''^^)"'^ . 
Note that the average number of check nodes which are not removed is^(l— (1 — (5)'^). Therefore, the normalized 
(by the number edges connected to the remaining check nodes) probability of an edge to remain is 



S+{l-5) [l-{l-Sf-') 



(9) 



1 - (1 - (5)fe 

If we substitute a(fc — 1)"^/^^"'^^ for (5 in (|9]) and use a Taylor expansion around fc = cx), we have e = i + o (i) . So 
the probability that a check node has degree t after removal is (J')e*{l-e)*~* which converges to Poisson distribution 
with mean 1 as k ^ oo. Following the approach used for the BEC case shows that the concentration theorem also 
holds for LMl on the q-SC. 

V. Scaling Laws Based on Stopping Set Analysis 

DE analysis provides the threshold below which the randomized (or non-uniform) recovery is guaranteed, in the 
following sense: the signal and the measurement matrix are both chosen randomly, and w.h.p. the reconstruction 
algorithm gives the correct answer. If the reconstruction algorithm is guaranteed to succeed for all signals of 
sufficient sparsity, this is called uniform recovery. On the other hand, if reconstruction algorithm is uniform over 
all support sets of sufficient sparsity, but succeeds w.h.p. over the amplitudes of the non-zero elements (i.e., has a 
small but non-zero failure probability based on amplitudes), then the reconstruction is called uniform-in-probability 
recovery. 

According to the analysis in section IIVI we know that the number of measurements needed for randomized 
recovery by using LM2-MB scales linearly with the sparsity of the signal. Still, the reconstruction algorithm may 
fail due to the support set (e.g., it reaches a stopping set) or due to the non-zero amplitudes of the signal (e.g., a 
false verification occurs). 

In this section, we will analyze the performance of MP decoding algorithms with uniform-in-probability recovery 
in the high-rate regime. This follows from a stopping set analysis of the decoding algorithms. A stopping set is 
defined as an erasure pattern (or internal decoder state) from which the decoding algorithm makes no further 
progress. Following the definition in [17], we let G = (y U C, i?) be the Tanner graph of a code where V is the 
set of variable nodes, C is the set of check nodes and E is the set of edges between V and C. A subset J7 C 1/ is 
a BEC stopping set if no check node is connected to U via a single edge. The scaling law below uses the average 
stopping-set enumerator for LDPC codes as a starting point. 
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A. Scaling Law Analysis for Stopping Sets on the BEC 

The average stopping set distribution Enj.k{s) is defined as the average (over the ensemble) number of stopping 
sets with size s in a randomly chosen {j, k) regular code with n variable nodes. The normalized stopping set 
distribution ^j^k{a) is defined as "ij.k{(^) — linin^oo ^ In £',i.j,fc(7ia). The critical stopping ratio a* ^ is defined 
as Q!*j, = inf{a > : ^j^kio) > 0}. Intuitively, when the normalized size of the stopping set is greater than or 
equal to a* k: the average number of stopping sets grows exponentially with n. When the normalized size of the 
stopping set is less than a*;,., the average number of stopping sets decays exponentially with n. In fact, there exist 
codes with no stopping sets of normalized size less than a* ^- Therefore, the quantity a* j, can also be thought of 
as a deterministic decoding threshold. 

The normalized average stopping set distribution 7j fc(a) for (j, fc) regular ensembles on the BEC is given by 
[18] 

lj.k{a)<-ij.k{a\x) = -\ri I '-^ \ -{j-l)h{a), 

where h{-) is the entropy of a binary distribution and the bound holds for any < x < 1. The optimal value xq is 
the unique positive solution of 

(1 + x)'^ — kx 

This gives the following theorem. 

Theorem 5: For any < 6* < 1, there is a iiT < cx) such that, for all k > K, a randomly chosen {j, k) regular 
LDPC code (j > 3) will (w.h.p. as n ^ oo) correct all erasure patterns of size less than 9ne{k — 

Proof: [Sketch of Proof] Since there is no explicit solution for xo, we use a 2nd order expansion of the LHS 
of ([Tol l around a; = and solve for a;. This gives xq — \J + o(a). Since 7j,fc(a) < 7j_fc(a,a;) holds for all 
a; > 0, we have 

7,,.(a) < I In ( ^'^^l^'^ ) -(j - (11) 

Next we expand the RHS of ( fTTT i around a ~ Q and neglect the high order terms; solving for a gives an upper 
bound on the critical stopping ratio 

'j-2-jln(A:-l)^ 



< exp 



J-2 

It can be shown that this bound on a* j, is tight as fc ^ oo. This means that, for any < 6* < 1, there is a if such 
that 9e{k - < a* ^ < e{k — f^j. ^jj ^ j{ Therefore, the critical stopping ratio a* f. scales 

like e{k - l)-J/0-2) as fc ^ oo. ■ 
Remark 5: Although the threshold is strictly increasing with j, this ignores the fact that the code rate is decreasing 
with j. However, if one optimizes the oversampling ratio instead, then the choice of j* = 2 + 21n(fc — 1) is 
nearly optimal. Moreover, it leads to the simple formula a*, ^ — and an oversampling ratio which grows 
logarithmically with k. 
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B. Stopping Set Analysis for the q-SC with LMl 

A stopping set for LMl is defined by considering a decoder where 5, T, U are disjoint subsets of V corresponding 
to verified, correct, and incorrect variable nodes. Decoding progresses if and only if (i) a check node has all but 
one edge attached to S or (ii) a check node has all edges attached to S UT. Otherwise, the pattern is a stopping 
set. In the stopping set analysis for q-SC, we can define En.j,k{o:, P) as the average number of stopping sets with 
|T| = na correctly received variable nodes and \U\ = n/3 incorrectly received variable nodes where n is the code 
length. 

The average number of stopping sets En.j^kio:, P) can be computed by counting the number of ways, Snj.kiO', b), 
that a correct variable nodes, b incorrect variables nodes, and n — a — h verified variable nodes can be connected 
to ^ check nodes to form a stopping set. The number Sn,j,k{(i, b) can be computed using the generating function 
for one check, 

gk{x, y)^{l + x + yt -ky- ((1 + xf - l) , 

which enumerates the number of edge connection patterns ("1" counts verified edges, "a;" counts correct edges, 
and "y" counts incorrect edges) that prevent decoder progress. Generalizing the approach of [18] gives 



n 



\Sn,j,k{an,Pn) 



\ na, n/3, n(l — a — P) 
En,j,k{a,P) = - r (12) 

\ nja, nj(3, nj{l -a- (3) ) 

where 

^„,,,fc(a,6) ^ coeff . 

For this work, we are mainly interested in largest /? for which En.j,k{o:, P) goes to zero as n ^ oo. Since the 
growth (or decay) rate of Enj^k{oi, /?) is exponential in n, this leads us to consider the normalized average stopping 
set distribution jj^k{ci,P) which is defined as 

7j,fe(Q!,/J) = lim - In En,j,k{a,p). (13) 

n—^oo n 

Likewise, the critical stopping ratio (3* j, is defined as 

= inf{/3 e [0, 1] : Wj,k{l3) > 0} (14) 

where 

Wj,k{l3)= sup 'yj,k{a,f3). 

ae[a,i-p] 

Note that Wj,k{P) describes the asymptotic growth rate of the number of stopping sets with number of incorrectly 
received nodes n(3. The number of stopping sets with size less than decays exponentially with n and the 

ones with size larger than nf3* ^ grows exponentially with n. 

Theorem 6: The normalized average stopping set distribution jj^k{a,P) for LMl can be bounded by 



July 30, 2009 



DRAFT 



17 



7i,fe(a,/3) <1j,k{a,P;x,y) ^ 
J^^ (l + jl + x + y)'^ -ky-jl + x)'') 

k j^koLykp 

+ (l-j)Ma,/3,l-a-/3) 



where the tightest bound is given by choosing (x, y) to be the unique positive solution of 

— ^ h — ^ 

1 + (1 + a; + y) - fcy - (1 + x)'' 

and 



(15) 



(16) 



y(^{l + x + y) 



l + {l + x + yf -ky-{l + x) 



k 



P- (17) 



Proof: Starting from (fT2l i and using Stirling's formula, it can be verified easily that 



1 , \na,nl3,n(l-a~l3) -Ni/ o i a\ 



lim 

nja.njl3.nj{l — a — l3 



where h{-) is the entropy of a ternary distribution. Using a Chernoff-type bound for Sn,j,k{o., b) (i.e., coeff (/(a;, y),x^y^ 
^^qjr for all X, y > 0), we define 

^■da B-x .^nj {lHl+x+yf~ky-{l+x)^) 

Minimizing the bound over x, y gives 

lj,k{a,l3) < jj^k{a,(3;x,y)^ 

■il;j^k{a,(3;x,y) + {1 - j)h{a, (3,1 - a- P), 

where (x, y) is the unique positive solution of ( fTST i and ( [TtI i. One can also show that the bound is exponentially 
tight in n. ■ 

C. Scaling Law Analysis for LMl Stopping Sets 

In CS literature, we are only interested in the scenario that j3 is small. It means we need to perform stopping 
set analysis in the high-rate regime or to the signal vectors with sparse support. For the convenience of analysis, 
we only derive the analysis for (j, fc) regular codes though it can be generalized to irregular codes [18]. In our 
analysis, the variable node degree j is fixed and the check node degree k is increasing. By calculating the scaling 
law of Wj^kiP), we find the uniform-in-probability recovery decoding threshold (3* f, which tells us the relationship 
between the minimum number of measurements needed for uniform-in-probability recovery and the sparsity of the 
signal. 
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Fig. 3. Numerical evaluation ui^ ^.(d) and theoretical bound v{d) 



The following theorem shows the scaling law of LMl for the g-SC. 

Theorem 7: There is a code from (j, fc) regular LDPC code ensemble and a constant K such that for the q-SC, 
all error patterns of size n& for 8 < j3j{k — ^.^jj recovered by LMl (w.h.p. as n ^ co) for k > K 

where /3j is the unique positive root on c of the following implicit function 



v{d) = -{{c-l)jln{l-c)-2cHc) 
+ (l + c)(j-2)(-l + lnd)) 



(18) 



where d ^ {1 - c)-J'/(j'-2)c2/0'-2). 

Lemma 4: Consider sequences of {xk,yk) given by (dSll and ^ which satisfy = 6 ((fc - l)-^/^-^)^ ^ 
goes to infinity. In this case, the implied Xk, yk, and ak all tend to zero. 
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Proof: See the Appendix IVII ■ 
Lemma 5: For the g-SC with LMl decoding and j > 3, the average number of stopping sets with size sublinear 
in n goes to zero as n — > (X). More precisely, for each 3 < j < k there exists a 6j^k > such that 

6=1 a=0 ^ ^ 

Proof: See the Appendix I VIII ■ 
Proof: [Proof of Theorem |7lThe main idea of the proof is to start from ( fTsT i and find a scaling law for 
Wj^kiP) as k grows. Since Wj,k{(3) is the exponent of the average number of stopping sets and the resulting 
scaling function v{d) is negative in the range (0,/3j), almost all codes have no stopping sets of size n6 with 
< S < i3j{k — Because finding the limiting function of the scaled Wj^kiP) is mathematically difficult, 

we first find an upper bound on wj^kiP) and then analyze the limiting function of this upper bound. 

Before we make any assumptions on the structure of x and y, we note that picking any x and y gives an upper 
bound of 7j fc(Q;,/3). To make the bound tight, we should pick good values for x and y. For example, the {x,y) 
which leads to the tightest bound is the positive solution of ( fT6] ) and ( fTTT i. Since we are free to choose the variables 
X and y as we like, we assume x and y decay with the same rate and are both equal to o (fe^r) ^^^^ Taylor 
expansions of ( fTSI l and SvH converge. 

Applying Taylor expansion for small x,y to ( fT6] l and (fTTj i, we have 

xy{k — 1) ss a 
{xy + y^) « /3. 
Solving these equations for x and y gives the approximations 



a P — a 

^0 ~ .,, =^ ya^' 



V(/3-a)(A:-l) Vfc-1 
Next, we choose a = c(3 for < c < 1, which requires that < a < □ Applying these substitutions to (flST l 
gives 



which equals 



(1 + c) (2 - j) (1 - ln(/3)) - (1 - c) J ln(l - c) 
- 2c hi(c) + (1 + c) J ln(-l + fc)^ + O . 

(19) 

Plugging /3 — d{k — jj^^^ ^^lis equation for d> gives 

^The scaling regime we consider is /3 = o{k~^) and this leads to the scaling of x, y. The scaling of x, y also implies that < a < /3. So 
we see that, although there exist stopping sets with a > f3, they do not occur in the scaling regime we consider. 



July 30, 2009 



DRAFT 



20 



7,,fe(a,/3) < ^(fc- l)--''/(-''-2)((c_ l)jln(l - c) - 2cln(c) 
+ (1 + c)(2 - - Ind)) + O ((fc - i)-2j/(J-2) j . 

(20) 

Scaling the RHS of (|20ll by (k - 1)^/0-2) gj^g^ jj^g limiting function 

v{c,d)^^{{c-l)jln{l-c) 
- 2c ln(c) + (1 + c) (2 - j) (1 - In rf)) . 

(21) 

Next, we maximize the scaled upper bound of •jj^kia, f3) over a by maximizing v{c, d) over c. The resulting function 
t'(ci) = maXcg(o,i) v{c, d) is a scaled upper bound on Wj,k{f3) as fc goes to infinity. Taking the derivative w.r.t. c, 
setting it to zero, and solving for d gives the unique solution 

rf=(l-c)-^/(^-2)c2/0-2). (22) 

Since the second derivative 

j \ ^^_^^_,y(,_2) 



2 V c 1 - 

is negative, so we have found a maximum. Moreover, v{d) is given implicitly by ( l2Tl i and ( |22] |. The only positive 
root of v{d) is denoted ;3j and is a constant independent of k. Fig.|3]shows the curves given by numerical evaluation 
of the scaled Wj^kiP), which is given by 



and the limiting function v{d). 

The proof is not yet complete, however, because we have not yet considered stopping sets whose sizes are 
sublinear in n. To handle these, we use Lemma |5] which shows that the average number of stopping sets with size 
sublinear in n also goes to zero. ■ 

Remark 6: In a CS system with strictly sparse signals and LMl reconstruction, we have uniform-in-probability 
reconstruction (w.h.p. as n — > oo) of all signals with sparsity at most nS where S < Pj{k— 1)^J/(J^2) jjjjg j-gquires 
TO = 771(5 measurements and an oversampling rate of 7 > 70 = f3j ^'^■' jj^^/j^ 

Remark 7: If the signal has all non-negative components, then the verification based algorithm will have no FV 
because the neighbors of a check node will sum to zero only if these neighbors are exactly zero. Therefore, the 
above analysis implies uniform recovery of non-negative signals that are sufficiently sparse. 
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VI. Information Theory and Sparse CS 

At first glance, the results in this paper seem to be at odds with existing lower bounds on the number of 
measurements required for CS. In this section, we explore the fundamental conditions for linear scahng using 
sparse measurements from an information theoretic point of view. 

Let k and j be check and symbol degrees; let n be the number of symbol nodes and m = be the number 
of check symbol nodes. The random signal vector X" has i.i.d. components drawn from fx{x) and the random 
measurement vector is Y^". The number of non-zero elements in the signal is controlled by assuming that the 
average number of non-zero symbol nodes attached to a check node is given by A. This allows us to write fxi^) = 
+ jfz{x), where Z is the random variable associated with a non-zero signal element. Since nj = mk, 
the condition oj < 1 implies k ^ oo and that the number of non-zero symbol nodes attached to a check node 
becomes Poisson with mean A. Therefore, the amount of information provided by the measurements is given by 



//(y™)<E^(^') 



1=1 



. oo , j 

= !^Ye-^^H(Z*Z-*---*Z) 
i times 

<jn'--f2e-'^iiH{Z)) 

i=0 

= jn^-^XH{Z). 

Since A/fc is the average fraction of non-zero symbol nodes, the entropy of the signal vector can be written as 

H {X^) = -nh (y) + njH{Z) 



This implies that 



= An^-" In + Xn^-"H(Z) + O (n^-^"^) . 



H{Y{^) - HiX]") < Xn^-'^ ( {j - 1)H{Z) ^ 



Xn 

Since a necessary condition for reconstruction is H (Y7") — H {X") > 0, we therefore find that 

f H{Z){j-l) + ln X' 
n < exp — ^ — — 

is required for reconstruction. This implies, that for any CS algorithm to work, either H{Z) has to be infinite or 
j has to grow at least logarithmically with n. This does not conflict with the analysis of LM2-MB in randomized 
reconstruction because, for signals over real numbers or unbounded alphabets, the entropy H{Z) can be infinite. 

VII. Simulation Results 

In this section, we provide the simulation results of LMl, LM2-MB and LM2-NB reconstruction algorithms and 
compare these results with other reconstruction algorithms. We consider two types of strictly sparse signals. The 
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Fig. 4. Simulation results for zero-one sparse signals of length 256 with 128 measurements. 

first type is the zero-one sparse signal where the entries of the signal vector are either or ±1. The second type 
is the Gaussian sparse case where the entries of the signal are either or a Gaussian random variable with zero 
mean and unit variance. We choose the signal length n = 256 and number of measurements m = 128. 

We compare different recovery algorithms such as linear-programming (LP) [23], subspace pursuit (SP) [33], 
regularized orthogonal matching pursuit (ROMP) [34], reweighted £p minimization (RWLP-p) [35], LMl, LM2- 
MB and LM2-NB. The measurement matrices for LMl, LM2-MB and LM2-NB are generated randomly from the 
(3,6), (4,8) and (5,10) ensembles without double edges and 4-cycles. We also pick the non-zero entries in the 
measurement matrices to be i.i.d. Gaussian random variables. In all other algorithms, the measurement matrices 
are i.i.d. Gaussian random matrices with zero mean and unit variancj^ Each point is obtained by simulating 100 
blocks. Fig. |4] shows the simulation results for the zero-one sparse signal and Fig. |5] shows the results for Gaussian 
sparse signal. From the results we can see LM2-MB and LM2-NB perform favorably when compared to other 

^We also tried the other algorithms with our sparse measurement matrices (for the sake of fairness), but the performance was worse than the 
dense Gaussian random matrices. 
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Fig. 5. Simulation results for Gaussian sparse signals of length 256 with 128 measurements. 

algorithms. 

Another interesting observation is that LMl, LM2-MB and LM2-NB are not sensitive to the magnitudes of the 
non-zero coefficients. They perform almost the same for zero-one sparse signal and Gaussian sparse signal. This is 
due to the verification-based nature of the decoding algorithm. The other advantage of LMl, LM2-MB and LM2- 
NB is the computational complexity is linear for both the measuring process (i.e., encoding) and the reconstruction 
process (i.e., decoding). 

We also find the maximum sparsity K* for perfect reconstruction when we use parity-check matrices from (3, k) 
and (5, k) ensembles (with different k) as the measurement matrices when n is large and try to see how K* scales 
with the code rate. In the simulation, we fix n = 10000, try different A;'s (or m's) and use LM2-MB as the decoding 
algorithm. Fig. |6] shows the how K* scales with m in high rate regime. We also show the theoretical scaling in 
Fig. |6] which is ctjuj /k with aj, = 1/6 and as w 0.37. Notice that the simulation and the theoretical results match 
very well. 

The simulation results for (3, 6), (4, 8) and (5, 10) ensembles are shown in Fig. |7]and Fig. [8] The results show 
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that for short block length and rate a half, using measurement matrix from ensemble with higher VN/CN degree 
leads to worse performance. This seems to conflict the results shown in Fig.|6] Actually, in the scaling law analysis, 
we consider rates close to 1 and large block-length which is not satisfied in the simulation of Fig. [T] and Fig. [S] 

VIII. Conclusion 

We analyze message-passing decoding algorithms for LDPC codes in the high-rate regime. The results can 
be applied to compressed sensing systems with strictly-sparse signals. A high-rate analysis based on DE is used 
to derive the scaling law for randomized reconstruction CS systems and stopping set analysis is used to analyze 
uniform-in-probability/uniform reconstruction. The scaling law analysis gives the surprising result that LDPC codes, 
together with the LM2-MB algorithm, allow randomized reconstruction when the number of measurements scales 
linearly with the sparsity of the signal. Simulation results and comparisons with a number of other CS reconstruction 
algorithms are also provided. 

Appendix I 
Proof of Proposition[T] 

Proof: Starting with the convergence condition A (l — e""^-'^) < x for x G (0, 1], we first solve for a.j to get 

olj = inf -— In (1 - \-^{x)) . (23) 
2;e(oa) ]x ^ ' 

Next, we substitute a; = A (1 — e~^) and simplify to get 

a - ^ inf — — ^ -. (24) 

For i > 3, this function is unbounded as y ^ or ^ oo, so the minimum must occur at an interior critical point 
y* . Choosing A(a:;) = x^^^ and setting the derivative w.rt. y to zero gives 



e 



-y 



= (25) 



f{i-e-y'f^-^ 

CanceUng terms and simplifying the numerator gives 1 — — (j — l)y*e~'y = 0, which can be rewritten as 
= (j — l)y* + 1. Ignoring y* = 0, this implies that y* is given by the unique intersection of e** and (j — l)y + 1 
for y > 0. That intersection point can be written in closed form using the non-principal real branch of the Lambert 
W-function [36], W-i{x), and is given by, for j > 2, 

y* = -j^ (l + ij - 1)W^, (^-_l_e-VO-i)^^ . (26) 

Using this, the a-threshold for j-regular ensembles is given by aj = iy* ^1 — e^^j ^ . For j = 2, the minimum 
occurs as yj and the limit gives S2 — ^. ■ 
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Appendix II 
Proof of Theorem[2] 

Before proving the theorem, we first prove the following lemma. 

Lemma 6: Let X be a Poisson random variable with mean A. The tail probability can be bounded by 

Pr(X>xo) <e-^(^)"""e-« (27) 
Proof: The moment generating function of Poisson distribution is Mx(s) = eM'^''~^\ The Chernoff bound 
shows 

Pr{X > xo) < e-'"=°Mx{s) 

for s > 0. Setting the derivative of the RHS over s to zero to minimize the RHS (since the second derivative is 
positive), we have the RHS of dZTl l. Note that the RHS of ( |27] | decays faster than e^^^" for any constant A. ■ 
Proof: [Proof of Theorem 2]To show the concentration for our case, we modify the proof as following. In the 
first iteration, we reduce the graph to a smaller one by removing the edges which are not erased. Let S = ^ be 
the channel erasure probability and k = n'^, the number of check nodes concentrates around ^ ~ jn^^'^ and the 
number of variable nodes concentrates around n' ^ nS ~ ajn^~'^. The probability that a check node has degree t 
after removal is - 5)*^"' which converges to Poisson distribution as fc ^ oo. So the edge degree distribution 

of check node is p{x) — e~"Ja:*~"'", the average check node degree is aj, and the edge degree distribution 

of the variable node is X{x) — x^^^. 

By the similar edge-revealing argument [16], we can model the graph-uncovering process as a martingale and 
apply the Azuma's inequality, we can bound the probability that the performance of a particular code deviates the 
cycle-free case by more than e. 



Pr 



^ ^ 



— — ' 

n' 



>,k<2e ^Er.T^ (28) 



where ai is an upper-bound of the difference of the numbers of erasure messages between the graphs obtained 
by revealing the {i — l)-th and the i-th edge, n'^ is the number of edges in the residual graph which concentrates 
around aj^n^"". p''^^^ is the fraction of erasures of the residual graph given by DE. It is easy to see n'p''^^^ = np'^^^ 
where p*^^^ is the DE result of the original graph. For simplicity, we can pick ai to be 2T^.i where ^ is the 
number of edges in the depth-2£ directed neighborhood of the i-th edge. Note that cti is identically distributed but 
not independent. So the law of large numbers does not apply in general. 

Since the check node degree is an i.i.d. random variable. By applying the union bound to Lemma|6] the probability 
that every check node in the residual graph has degree no more than cn'^ can be bounded as follows 

Pr(every CN has degree < n"') < n'e"^"'" < e"^'"'" (29) 

for sufficiently large n' , where A' ^ A — e. 

For a (j, n'^) regular ensemble, Te^i = Tf = Q {n'^^) and — Q (n'^^^) . So we have 

{n'jef ^ B'{n'ief _ „ ,„/i-2fe 



< ' ' = -Be^n' 



2 ^"^(^i+j) ^2 ~ 2n'jn'2^« 
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given every check node has degree no more than n'^ where B' and B are constants independent of n' . Note that 
the event that — p'^^-*] > e can be caused by two reasons. The first one is the rare events that captured by 
Azuma's inequality. The second reason is that some check nodes have very large degrees such that the RHS of 
dU has positive exponent. Equation (|28] | shows the probability that | — p'^^^ \ > e which is caused by the first 
reasons and ( |29l ) shows the probability that | — p'^^^ \ > e which is caused by the second reason. Note that the 



event 



p'W I > £ can be caused by both of the first and the second reasons. By the union bound, 



Pr 



(30) 



Now we set 1 — 2£6 = 9 to balance the probability that Azuma's inequality fails and the probabiUty that any check 
node has too large degree. This gives 6 = yzpgi- '^^ (l30l t can be rewritten as 



Pr 



An 



> e \ < n'e 



n e 



where (3' is a constant independent of n. 



Appendix III 
Proof of Lemma[2] 

Proof: All statements are implied to hold for all fc > S^-^^, all X e [0, 1], and all a e [0,aj]. Since l — (l — a;)'^ 
is concave for A; > 1, the tangent upper bound at x = shows that 1 — (1 — x)'' < kx. This implies that 



1-1- 



< 



(31) 



Since j^j/u-i) < < 1' we can use dSTT i to upper bound gk+i{x) with 



9k+iix) < — 1 
a, \ 



— J — 1 7 — 1 



— J — 1 7 — 1 

X-' 

k 

-\ k\ 3-1 



a / 

< ^ 1 
a, \ 



— 3 — 1 7 — 1 

x-i 



-\ 3-1 



This completes the proof of (i). 

The fact that 'gi.^i{x) is monotonically decreasing follows from Lemma [T] This completes the proof of (ii). 
Lemma [U also shows that the limit of is 

.*(-)^^(i-e-rv-y-\ 

This proves the first part of (iii). 
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Next, we will show that 



lim gk{x) = — 1 - e 



Fkst, we show that 



lim fc I 1 - ( 1 - 



— J-l 7-1 



(32) 



In hght of the the upper bound (l3Tl i. the limit is clearly upper bounded by ^. Using the lower bound in 

Lemma [H we see that 



1 - 



> 



(1 + ajXfc--J/0-i)) ' 



> 



(1 +SjXfc-J'/0'-i)) 



> 1 



This implies that 



Together with 



1-1 



fcj/(j-i) 



> 1-1 



J-1 



e ' 



— i-i i-i 



we see that the limit ( |32] ) holds. 

To calculate the limit of gk{x), we can use the fact that limfc^oo (l — at + o (■!:)) = g-iimfc^oo feifc whenever 
limfe^oo fcflfc exists. Using this, we see that limfe^oo 9k+i{x) can be rewritten as 



lim 



1- ( 1- ( 1 



j-i 



1 — e 3 



where the last step follows from 

Appendix IV 
Proof of Lemma[T] 

Proof: Recall that aj is defined as the largest a s.t. I 1 — e^" ^ j < x for x G (0, 1]. So aj can be 
written as 

aj ~ inf (33) 

Ke(o,i] 

where = (— hi (l — a;^/'--'^^-') x^^^-'-') . Notice that ft.j(a;) is a monotonically increasing function of x 

when = 2. So we have 

OLi = lim = 1. (34) 
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When j > 3, hj{x) goes to infinity when x goes to either or 1, so the infimum is achieved at an interior point 
X*. By taking derivative of x and setting it to zero, x* is the solution of 

--ij- If ■ (35) 



1 — XJ-i I In I 1 — XJ-i 



So 



^: = 1 + ...... > — ... ..ox ■ (36) 



(j-l)2T4-_i(-e-i/a-i)V(j-l)2)^ 
By solving this numerically, we find that xl « 0.816042, xl « 0.938976 and xl « 0.971087. Substituting x* into 
(|33]l, we have as « 1.87321, w 1.66455 and as w 1.52073. ■ 

Appendix V 
Proof of Lemma[3] 

Proof: Let us define the function gk{x) with 



'9k{x) = 



+ 0-1)1-1 



1-1 



ajx 



ajx 
k-i\ 



k-i\ -J'-i 



ajx 



k-l 



To prove (i), we will show gkix) < gk{x) < gk{x)- To see that gk{x) < gk{x), we must simply observe that 

This can be seen by working from the inner expression outwards and using the facts that < ^ < 1 and < x < 1 . 
Each step gives a result that is bounded between and 1 . 

k 

To show gk{x) < gk{x), we first change variables to z = (l — where z E (0, 1). This allows gk{x) to be 

written as a function of z with 



-gk{z) = ^ ((1 - zy-' + U - 1) (1 - zf-' z) . (37) 

Taking the derivative of gk{z) with respect to z gives 

%^ = -^0-2)0-l)(l-z)-^z (38) 
az aj 

which is negative for j > 3. So gk (z) is a monotonically decreasing function of z. Using the inequahty (l — > 
(1 - ^)'^ , we find that gfe(x) < gkix). 

Next, we will prove (ii) by showing the hmits of gk{x) and gk{x) are the same. Fkst, we take the the term by 
term limit of gk{x) to see that 
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lim 5fc(x)=^((l-e-"^^)' ' 



fc— >oo (y.j 



{j - 1) (1 - e-"J'^)^' ^ e-"J'^) 



Next, we use the fact that (l - ^)''~^ e""-''^ to see that 



(1 - e-°^^)^' ^ (1 + (j - 2)e-"^'^) . (39) 



fe^oo \ 1 - ^ \ V k 

* k \ ^ 



/c-l\ J ^ 



To prove (iii), we recall that, using the change of variables z ~ (l — , gk{z) is a monotonically decreasing 



From this, we find that the term by term limit of gk{x) is also equal to ( [39] l. 

k 
k 

function of z. Moreover, gk{z) does not depend on k and z ~ (l — is a monotonically increasing function 

of k (e.g., see Lemma [T]!. So gk{x) is a monotonically decreasing function of k. ■ 

Appendix VI 
Proof of Lemma [4] 

Proof: Consider whether the sequences Xk and yk converge to zero or not. Clearly, there are only 4 possible 
cases. 

If Xk = o(l) and yk — the limit 

hm (3k = lim — — -r — = lim yk (40) 

fe— »00 fe-+00 (^i + yk) fc— >CXD 



contradicts f3k = Q {{k - 1)^1/(3"^)) . 
If Xk = and yk ~ o(l), the limit 



lim kf3k — lini — — — ''\_^ = 1 
fc~*oo ''^^^ yk{l + xk)' 



contradicts (3k ^ & {{k - 

If Xk = and yk ~ the limit satisfies 

> lim — , 

k^oo 1+ Xk +yk 

and this contradicts /3fc = 6 ((fc - l)-^/^^-^)-)^ ^ 

Appendix VII 
Proof of Lemma |5] 

Since all stopping sets with size sublinear in n shrink to the zero point on the scaled curve, we must treat sublinear 
stopping sets separately. The proof proceeds by considering separately stopping sets of size O(lnn) and size 6n 
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for very small 5. The number of correct and incorrect variable nodes in a stopping set is denoted, respectively, a 
and b (i.e., na = a and n/3 — b). Proof: Using (fT2] i and Lemma |71 we can bound Enj^k{oi, /3) with 

i?«,,.fc(a,/3)<jeT2We(i-J>''("''5'i-"-'^)5„,,-fe(an,/3n). 

The coefficient Sn,j,k{0', b) can be bounded using a Chernoff-type bound and this gives 

lny„j,fc(a,6) < — In- 



k xi'^y^^ 



< ((1 + x + y)'' — ky —kx) —jalnx—jblny 

k 



for arbitrary x > and y > 0- Choosing x — and y — gives the bound 

<C'„(a+6b72^ (41) 

where C is a constant independent of n. Applying dTTJ to the £'„j.fc(a, (i) bound shows that Enj^k ^) 

< jeTsb exp f (1 - j)nh [-,-,1----]] 5„.j- fc(a, b) 

< jeWCn('^+'')(^/^-(^-i)(i-)) {-\ ''""^ f (42) 

where < e < |; and j > 3. 

Now, we can use this to show that 

A Inn n — 6 / , \ 

6=1 0=0 ^ ^ 

Since a stopping set cannot have a check node that attaches to only verified and correct edges, a simple counting 
argument shows that Sn,j,k{a, b) — if a > {k — 1)6. Therefore, the above condition can be simpUfied to 

yllnn(fe-l)b X ,s 

lim V V En.j,k{-,-] =0. (43) 

b=l a=0 ^ ^ 

Starting from (|42ll, we note that 6 < Alnn and a < (fc - 1)6 implies that (^)^^^^'"' (;^) < 1 for large 
enough n. Therefore, we find that the double sum in ( |43] | is upper bounded by 

for large enough n. Since the exponent (a + b){j/2 — {j — 1)(1 — e)) of n is negative as long as e < ^ and j > 3, 
we also find that the limit of the double sum in (l43T l goes to zero as n goes to infinity for any ^ > 0. 

Now, we consider stopping sets of size greater than A Inn but less than dj.kn. Combining (fTSl l and Lemma [T] 
shows that En,j,k{a, [3) < je^^ e'^^^-*'^"'^^\ Notice that (fT9] l is an accurate upper bound on jj^kia,P) for small 



July 30, 2009 



DRAFT 



31 



enough (3 and its maximum over a is given parametrically by ( fTSl ). Moreover, v{d) is strictly decreasing at c? = 0, 
and this implies that 7j,fc(a, /3) is strictly decreasing in /3 at /3 = for all valid a. Therefore, there is a 6j,k > 
and ?7 > such that 

7j,fc(a,/3) < -77/3 

for all < /3 < Jj.fc. From this, we conclude that < [3 < Sj^kn which implies that 



where Ary can be made arbitrarily large by increasing A. Choosing A = - so that Arj = 3 shows that 



b— log n a—0 



1™ Yl Yl ^n.j,k (-,-)< lim n^jeii" n ^ = 



n n J n^oo 



This completes the proof. 



Lemma 7: The ratio D = 



n 



a,b,n — a ~ b 



Appendix VIII 
Lemma|7] 

\ 

/ 



can be bounded with 



nj 



^ aj, bj, {n-a- b)j ^ 



J exp (1 - j)nh{-, -,1 )- \<D 

n n n n IZn / 

I ^ 7 / « ^ o, b 1 
< jexp {l-j)nh{-, -,1 ) + 



n n n n i2jn 



Proof: Let D be defined by 




Using StirUng's approximation, the binomial coefficient can be bounded using 
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: exp ( nh{X) — 



V27rnA(l - A) V 12nA(l - A) 

- V2.nA(l-A) ^"P^"'^'^^' 




where /i(-) is the entropy function in nats [37]. Applying this bound to D gives, after some manipulation, that 



jexp(^(l-,)(n/.(^,l-^) + 

\a + b a + bj J 12nJ 

( , J , fa. + b ^ a + b\ 



Finally, we notice that 



nh 



b a + 6 



n 



+ (a + b)h 



ba + b 



nh 



a b ^ a b 
n n n n 



This completes the proof. ■ 
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Fig. 6. Simulation of high rate scaling of (3, k) and (5, fc) ensembles for block length n = 10, 000. 
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Fig. 7. Simulation results for zero-one spikes of length 256 with 128 measurements by using (3, 6), (4, 8) and (5, 10) ensembles. 
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Fig. 8. Simulation results for Gaussian spikes of lengtli 256 witli 128 measurements by using (3, 6), (4, 8) and (5, 10) ensembles. 
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